Communications architecture for a high throughput storage processor

ABSTRACT

A storage processor particularly suited to RAID systems provides high throughput for applications such as streaming video data. An embodiment is configured as an ASIC with a high degree of parallelism in its interconnections. The preferred embodiment provides a store and forward architecture configured around a switch with prioritization on data pathways critical to high throughput.

BACKGROUND

[0001] Computer systems and networks require increasing quantities ofstorage capacity. The need for large amounts of storage together withconcerns about reliability, availability of data, and other issues haveled to the development of centralized file servers that store files fora number of users. File servers are usually managed by a single entityfor many users providing at least the obvious economies for IT managers.In file servers, many of the benefits that accrue in terms ofreliability and availability arise because many individual hard disksare used in concert. File servers that seek to maximize these advantagesthrough a variety of mechanisms fall under the umbrella term “RedundantArray of Independent Disks” or “RAID.” The operations of individualdrives in a RAID array may be coordinated to provide fast reliableservice to many users with a minimum of burden on the systems supportedby it, such as networks or individual hosts attached to it.

[0002] In a typical RAID configuration, data is divided among the drivesat the bit, byte or multiple-byte block level. This leads to thepossibility of reading and writing data to multiple drives in parallelfor even individual file requests. Because the speed of disk drives isusually the bottleneck in file server systems, not the memory or busspeeds, such parallel access can lead to manifold increase inthroughput. In fact, throughput enhancement due to this parallel accesscapability is a commonly-enjoyed benefit of RAID storage systems.

[0003] The general technique used to divide data among multiple drivesis called data “striping.” The reason for the term “striping” is that asingle stream is written in uniformly-sized blocks in a regular sequencethat can be depicted diagrammatically is painting a stripe across anarray of disks. Generally, in these systems, reliability advantages areobtained by adding redundancy to the stored data so that if a disk drivegoes bad, the stored data can still be retrieved intact from the datawritten to other disks. Many types of self-correction systems are knownin which even a small amount of additional data can be used toreconstruct a corrupted data sequence. A simple way of doing this iscalled mirroring where the same data is duplicated on more than onedisk. A more sophisticated technique (and one that is highly prevalent)is to generate a relatively small quantity of parity data, which can beused to reconstruct a bad data stream. Often mirroring and striping areused in concert so that each “drive” is actually a small mirrored arrayconsisting of more than one drive containing copies of the same data.

[0004] Significant processor capacity is required in RAID systems. Thegeneration of the redundant data used for regenerating good data when adrive goes bad is computationally intensive. Whenever new data iswritten to a RAID array, parity data must be generated for each block ofdata. Parity computation uses a logical operation called “exclusive OR”or “XOR”. The “OR” logical operator is “true” (1) if either of itsoperands is true, and false (0) if neither is true. The exclusive ORoperator is “true” if and only if one of its operands is true. Itdiffers from “OR” in that if both operands are true, “XOR” is false. Thereasons this operation is used is that it is pretty easy to design aprocessor that can do a lot of XORs very fast. Also, when data is XORedtwice, it is undoes the first XOR operation. XOR A with B and then XORthe result with B and the result is A. In RAID, parity data is generatedfrom the data to be stored by XORing each data block with the next datablock in a succession, say, of four data blocks. Then if the four blocksare written to four consecutive disks and the result of the four XORoperations written to a fifth (redundant) disk, when one of the fourdrives goes bad, XORing the remaining good blocks of the four and thefifth will generate the bad block. The operation works for any sizeblocks. When data is read from the array, XOR calculations may or maynot be performed depending on the design of the system.

[0005] There can be a big difference between read and write performance.Random reads may only require parts of a stripe from one or two disks tobe processed in parallel with other random reads that only need parts ofstripes on different disks. But for random writes, every time data onany block in a stripe, is changed, the parity for that stripe has to becalculated anew. This requires the writes not only for the particularblocks to be written, but also reads of all the other pieces of thestripe plus the writing of the new parity block.

[0006] The generation of all the parity data takes a substantialquantity of processing capacity in both the read and write directions.Also, the reconstruction of data read from a bad disk takes a great dealof computation. When a disk goes bad, the hardware has to be fixed, butthis is preferably done without halting access to data and the systemgoes into what is called “degraded mode” or “degraded state,” continuingto operate, but without all of its disks. Modern RAID systems aredesigned so that bad data retrieved from a corrupted array is recognizedand corrected and delivered to users in such a degraded mode. But thisdegraded mode is normally significantly limited in some ways that may ormay not be apparent to a given user. Users requiring high throughputrates, particularly, may notice a significant loss in performance. Forexample, users requesting large files, for example streaming audio orvideo, may find degraded mode provides significantly slower performance.The construction of parity data and the processes of error checking anddata reconstruction are not the only computational burdens for a RAIDarray. Modern disk array architectures also perform operations such asrequest sorting, prioritizing, combining and redundancy management,among others.

[0007] The computer hardware behind RAID systems are usually fairlyspecialized systems, designed to maximize the performance of a verynarrow range of processes, such as queuing, error detection, errorcorrection, and high speed data transfer including caching. Becausethese requirements are fairly specialized, the hardware designed tosupport high performance RAID systems has generally been ratherspecialized. Typically these design requirements are based onperformance benchmarks involving many simultaneous accesses to smallfile units. For example, performance might be indicated by input-outputoperations per second based on an average 2-kilobyte file size. Thistype of benchmark places more emphasis on latency and less on throughputand, as a result, typically RAID systems are designed to emphasizethese. However, as mentioned above, streaming data places very differentdemands on a storage system and these are particularly difficult to meetwhen the system is operating in degraded mode. Systems designed toperform well in terms of the traditional benchmarks tend to do ratherpoorly in such situations. Thus, there is a need for storage systemdesigns that are capable of providing high minimum performanceguarantees for throughput under degraded mode operation.

SUMMARY OF THE INVENTION

[0008] A storage processor for a block storage RAID array services diskstorage block requests from one or more hosts. At its heart, aapplication specific integrated chip (ASIC) supports a store and forwarddata transfer regime in that host to disk transfers are made by placingdata in storage processor memory under control of the storage processor,operated on by the ASIC, and sent to the disk array. Disk to hosttransfers are made by placing the same data store, checked orregenerated by the ASIC, and sent to the requesting host. The main datahighway in this model is a host-memory-disk path and memory bandwidth istherefore critical. A single memory space is addressed, in the preferredembodiment, by multiple buses under software management to even the loadand provide bandwidth approaching a multiple of that of a single bus.

[0009] The problem of achieving high throughput, even under degradedmode conditions, is addressed by providing parallel execution of certainoperations that are identified, in the context of the chosenarchitecture, to be critical to a minimum throughput guarantee. To cleara path for parallel execution, coherence issues that would normallyarise with caching are avoided by relying on a cacheless configuration.Point-to-point communications are defined using switches and a number ofFIFOs in key ways that allow the sharing of channels between the variousdevices transferring addressing and user data. By avoiding the use ofcaches altogether, data traffic to support coherency (e.g. broadcastingof invalidates and such) is eliminated. In addition, the controlprocessor or processors need never operate on the data transferredbetween the host and disks, allowing parity calculation and datatransfers to be handled with a minimum burden on the control processor.

[0010] The control processor is further unburdened by providing parallelexecution of data transfers and parity calculations with prioritizationand programming being prompted by via interrupts. Efficient handling ofordering is, preferably, provided by hardware logic-based masking ofinterrupts and by other mechanisms described further below.

[0011] The invention or inventions will be described in connection withcertain preferred embodiments, with reference to the followingillustrative figures so that it may be more fully understood. Withreference to the figures, it is stressed that the particulars shown areby way of example and for purposes of illustrative discussion of thepreferred embodiments of the present invention or inventions only, andare presented in the cause of providing what is believed to be the mostuseful and readily understood description of the principles andconceptual aspects of the invention or inventions. In this regard, noattempt is made to show structural details of the invention in moredetail than is necessary for a fundamental understanding of theinvention or inventions, the description taken with the drawings makingapparent to those skilled in the art how the several forms of theinvention or inventions may be embodied in practice.

BRIEF DESCRIPTION OF THE DRAWINGS

[0012]FIG. 1A is a figurative representation of a storage processoraccording to the prior art.

[0013]FIG. 1B is a figurative representation of an environment in whichthe storage processor of various embodiments of the invention may beused.

[0014]FIG. 2A is an illustration of a prior art storage processor devicein which a control processor was used to control a DMA, but which wasnot involved directly in processing user data, thereby liberating theprocessor of performing the calculations necessary to reconstructsectors from parity data, perform error checks, and keep informed ofchanges to user data.

[0015]FIG. 2B is a diagram illustrating parallel processes for multiplethreads to show how throughput can approach bus speed even withreconstruct and parity generation.

[0016]FIG. 2C is a another storage processor device supporting featuresof the present invention which has the same advantages as the device ofthe prior art, but which is capable of providing faster throughput andis more expandable.

[0017]FIG. 3 is a figurative representation of a storage processoraccording to a preferred embodiment of the invention.

[0018]FIG. 4 is an illustration of a storage processor showinginterfaces to requesting processes, storage, and control processordevices.

[0019]FIG. 5 is an illustration of various details of a storageprocessor showing data path components, memory, and control devices.

[0020]FIGS. 6A and 6B illustrate data path features incorporating alook-aside buffer that may be included in various embodiments of theinvention.

[0021]FIG. 6C is a detailed illustration of an embodiment of a storageprocessor for purposes of illustrating the data path features of FIG. 6Band other storage processor features according to embodiments ofinventions disclosed.

[0022]FIGS. 7A and 7B illustrate the processing of a write requestaccording to embodiments of inventions disclosed.

[0023]FIGS. 8A and 8B illustrate the processing of a read requestaccording to embodiments of inventions disclosed.

[0024]FIG. 9 illustrates a technique for determining when a data path isflushed of data relating to an interrupt.

[0025]FIG. 10 illustrates a process of writing data to a BE store fordiscussion of I/O parallelism features of embodiments of inventionsdisclosed.

[0026]FIGS. 11A and 11B illustrate a multiple-threaded set of processesof writing data to a BE store for discussion of I/O parallelism featuresof embodiments of inventions disclosed.

[0027]FIGS. 12A and 12B illustrate a multiple-threaded set of processesof reading data from a BE store for discussion of I/O parallelismfeatures of embodiments of inventions disclosed.

[0028]FIGS. 13A and 13B illustrate multiple threads of various processesfor discussing I/O parallelism features of embodiments of inventionsdisclosed.

DETAILED DESCRIPTION OF THE INVENTION

[0029] Referring now to FIG. 1A, a storage processor 15 provides variousprocesses to support the access, by one or more requesting processes 10to data storage in the form of some kind of media devices 17. In anembodiment, the requesting processes 20 are one or more host computersor network nodes and the media devices includes multiple disks in aredundant array (of independent disks; RAID). The requesting processesstore and retrieve data, at least some of which may be stored on themedia devices 17. The supporting processes provided by storage processor15 may include data security, data compression, error detection, errorcorrection, address translation (from a type employed by the requestingprocess to a type employed by the media devices), and processes thatincrease the speed of access for reading and writing from/to the mediadevices 17.

[0030] The requesting processes 10 may include ones operating on asingle computer or many. The communication between the storage processor15 and the requesting processes 10 may take any form used for datacommunications including any protocol or physical media devices such asused for networking, satellite communications, intradevice backplanecommunications such as buses, or any other. The same as true of thecommunications interface between the storage processor 15 and mediadevices 17. The storage processor 15 may provide for translation ofprotocols to/from one or more employed by the requesting process(es) andto/from one or more employed by the media devices 17.

[0031] Although not depicted explicitly in FIG. 1A, the storageprocessor 15 may include one or more similar devices interconnected bycommunications channels to provide redundancy as illustrated in FIG. 1B.The same is true of the media devices 17, which may include, forexample, mirrored disks or arrays of disks with identical data such thatif one needs to be replaced, its role can be played by its mirror drive.

[0032] Referring now to FIG. 2A, a prior art storage processor devicehas a control processor 21 to control a DMA/XOR engine 25. The controlprocessor 21 is not involved directly in processing user data, therebyliberating the processor of the burden of performing the calculationsneeded to generate parity data, reconstruct sectors from parity data,perform error checks, and keep informed of changes to user data. Data iscarried by a first bus 34 between a front end (FE) interface connectedto requesting processes 10 and a back end (BE) interface connected tomedia devices 17 through write and read memories 32 and 33. User data istransferred between the FE interface 28 and the write memory 32. Thedata is operated upon (e.g., CRC generated, addition of parity blocks,time stamps and other meta-data, etc.) by the DMA/XOR engine 25 andtransferred to the read memory 33. The DMA/XOR engine 25 then transfersthe user data from the read memory 33 to the BE interface 29. Thecontrol processor 21 issues commands through a bus 22 via a bus bridge24 and bus 34 b to the DMA/XOR engine 25, which is controlled by a DMAprocessor 26. Bus 34A carries control data from the FE and BE to theProcessor 21. Bus 34C carries user data. The user data and control datain the device of FIG. 2A separated user data and control data so thatthe processor 21 and the processor bus 22 was not burdened byinformation relating to changes in user data stored in read and writememories 32 and 33. The FE- and BE-attached devices (requestingprocesses 10 and media devices 20, for example) communicate theirrequests to the control processor 40 by storing requests for actions inthe control memory 23 and interrupting the processor. The respectiverequester may transfer user data to a respective memory (32 or 33)independently. The processor then generates instructions in queueslocated in control memory 23 which the DMA processor 26 accesses andexecutes.

[0033] User data going from FE to BE or vice versa can be moved into arespective memory and read and processed in parallel processes so thatdata can move at speeds approaching the capacity of the buses (34C, 34D,and 34E). This can be true even though the user data may need to bereconstructed or when new parity blocks have to be generated as new datais written to the BE. However, this assumes there is no need forsimultaneous bidirectional traffic between the DMA/XOR engine 25 and XORFIFO 27 and one of the read or write memories 33 and 32. This is becausethe transceivers 19 and 18 enable transfers of user data in only onedirection at a time.

[0034] Referring now to FIG. 2B, arrows represent continuous streams ofdata between FE, BE, and DMA/XOR processes for multiple simultaneousthreads all of which are assumed to be writes of new data includingsmall block transfers for which both old (from the BE) and new data(from the FE) are required for generation of non-longitudinal paritydata. User data transfers from the FE interface 28 and write memory 32are represented by the arrow A. User data transfers from write memory 32to read memory 33 are represented by arrow B. Arrow C represents datasnooped by the DMA/XOR engine 25 and used for parity calculations withthe XOR FIFO 27. Arrow D represents the results of parity calculationsto be transferred to the BE and arrow E, the data to be transferred,including the stream represented by D, to the BE. The arrows F andrepresent old data required for calculating new parity sectors whichmust be transferred from BE through read memory 33. Note that there isopportunity here for significant I/O parallelism. However, since thetransceiver 18 is present, it is not possible for data to besimultaneously read into the XOR FIFO 27 from read memory 33 and out ofthe XRO FIFO 27 into read memory 33. Even though programming mayinterleave the processes of multiple threads, some degree of attenuationof throughput due to the inability to transfer data from the bus 34E and34C bidirectionally.

[0035] Referring now to FIG. 2C, in a different storage processordevice, the DMA and XOR engine are combined in an ASIC with a crossbarswitch that links FE and BE buses 34 and 35 with a single user memory39. A FE interface 28 communicates with a control memory 42 throughbuses 34 and 36 bridged by an I/O bridge 43. A BE interface 29communicates with the control memory 42 directly by way of the bus 35,which is connected directly to a memory controller and I/O bridge 37.Only control data is carried through the I/O bridges 43 and 37. In thisembodiment, a single memory 39 is used instead of separate read andwrite memories 32 and 33. This permits user data to be moved in a singlehop between FE and BE. User data that does not need to be translated ormanipulated by the DMA/XOR engine 25 need not be written more than onceon its way between FE and BE, which reduces the data load on the memorychannels (buses for example). Although the read and write memories 33and 32 of the device of FIG. 2A may be capable of simultaneous read andwrite from the same memory, the write channel of each must be burdenedwith the same user data (although possibly altered by the addition ofmeta-data). The use of read and write memories 33 and 32 created a storeand forward configuration, which enabled parallelism because the FEcould transfer to/from the write memory 32 while the BE transfersto/from the read memory 33 simultaneous with transfers from write memory32 and read memory 33 with snooping by the DMA/XOR engine 25. Incontrast, in the configuration of FIG. 2C, which shares many commonfeatures with preferred embodiments discussed in connection with FIGS.3-12, data is written to memory 39, operated on as appropriate, and thenread from memory 39. As is discussed with regard to FIGS. 10-12, thiscan be managed in such a way as to provide a high degree of parallelismand ensure high throughput of user data. As discussed above, this isadvantageous for large file transfers with a minimum guaranteedbandwidth such as streaming data.

[0036] Referring now also to FIGS. 3 and 4, in a preferred embodiment,the storage processor 15 may include a FE interface 30 to supportconnection to the requesting processes 10 and a back end (BE) interface48 to support connection to the media devices 17. A processor end (PE)interface may support connection to a control processor 45. A backplanecommunications device 50 may serve as a data conduit for supportingcommunications among the devices shown. A memory 55 may provide databuffering and other temporary data storage for supporting devices shown.A data controller/translator 60 may govern specific data transfer andtranslation operations, which may involve the calculation or processingof error checking and/or error correcting data. The datacontroller/translator 60 may execute its functions in response tocontrol commands from control processor 45 connected through the PEinterface 49. The processor may be connected through a memory controllerhub (MCH) 47 and through it address its own dedicated control memory 46.

[0037] The backplane communications device 50 may consist of one or morebuses, switches such as crossbar switches, direct connections, orcombinations thereof. Together, the backplane communications device 50defines a set of communications channels, which may include buffering,data translation, and control operations such as arbitration. The FEinterface 47 may include temporary storage such as buffers, controllogic, and other functions and is preferably substantially identical tothe BE interface 48. Not shown explicitly, although they may be includedwithin or external to the interfaces 47, 48 and 45, are: protocolsupport for the connections to the external devices. For example, asillustrated in FIG. 4, an interface for an external bus such as PCI-X 70or a network adapter such as an Ethernet or Infiniband (not shown) maybe provided and further adapters for connecting to the host or media maybe provided. For example, the interface to the FE and BE interface couldinclude a PCI-X interface 70, 80, a PCI-X bus 72, 82, a PCI-X to hostadapter 74, 84 for the external protocol/media 76, 86. The latter may,for example, act as a bus master and protocol translator. The PEinterface may be connected to another external adapter, for example,PCI-X by a PCI-X interface 90, a PCI-X bus 92 to which the processor isconnected. In the above example, the PCI-X 70, 80, 90 interfaces orother interfaces on the storage processor 16 side of the bus could beintegrated in a single device, such as an ASIC.

[0038] Physically, parts of the interface could be provided on the samedevice as the components shown in FIG. 3 or externally. For example, thecomponents plus the interfaces (not shown) could be burned on a singlechip (ASIC) or programmed on a field programmable gate array (FPGA). Thememory 55 could include any type of high speed memory subsystem such asone or more RAMBUS channel.

[0039] In a preferred embodiment of the invention, the backplanecommunications device 50 provides parallel data paths between selecteddevices as discussed below. Referring now to FIG. 5, the PE, FE, and BEinterfaces 100, 135, and 140 of a preferred embodiment represented morefiguratively in FIG. 3 are connected via one or more crossbar switchesand/or direct connections represented by control data communicationsdevice 110 and a user data communications device 115. In the preferredembodiment, structurally, the latter represent a switch or other devicecapable of providing parallel or high bandwidth communications betweenthe devices, preferably in parallel. Also, some communications channelsmay be provided by direct links because of a lack of contention or otherconsiderations such as economies in fabrication. In FIG. 4, the factthat separate data communications devices for user and control data 110and 115, is not intended to indicate that it mirrors a structuralfeature. Both may be subsumed in one switch or other device or mayresult from the combined behavior of multiple devices.

[0040] In the preferred embodiment of FIG. 5, the datacontroller/translator 60 may correspond, at least in part, to a DMA/XORengine 125 and the memory 55 to memory 130. The local space 120 is anabstraction of any semaphores, temporary data stores, hardware orsoftware state machines, etc., which provide information to the controlprocessor 45 and are primarily accessed by it. Broken lines asindicated, for example, at 170 indicate channels permitting the transferof address information. Solid lines as indicated for example at 175indicate channels permitting the transfer of data. The latter lines havearrows to show the direction in which the respective classes of data mayflow. The solid lines appearing in pairs with double arrowheads, asshown, for example, at 165, indicate channels permitting two-waytransfer of data. Figuratively speaking, the control data communicationsdevice 110 and the user data communications device 115 is an abstractionwhich represents any suitable communications fabric permitting, at leastto some degree, simultaneous two-way transfer of data from one channelto another connected channel.

[0041] The two-way data channels (e.g. at 165), as said, may permitsimultaneous data transfer in both directions. The bandwidth may be anysuitable, but in a preferred embodiment, each channel is a switchedparallel path that is 64 bits wide. The addressing lines may provide anychosen bandwidth, but in the preferred embodiment, they are switchedparallel paths 32 bits in width.

[0042] Referring back to FIG. 1A, requesting processes 10 make requeststo write and read data to and from the media devices through the storageprocessor 15. The requests may include address data and user data, thelatter including the data that is to be stored, if the request is for awrite to the media devices. Normally, the requesting process 10 requestscontain no information relating particularly to details of the mediadevices, which may include, for example, magnetic disks in a RAID array.That is, the addressing scheme employed on the requesting process 10side of the storage processor 15 may uniquely point to the particulardata content being requested, but have no relevance in terms of wherethe data is stored. The storage processor 15 must translate from theaddress provided by the requesting process 10 and the addressing schemeused to identify the location or locations where the data is stored inthe physical media devices 17. The address translation process caninvolve substantial machinations, particularly because a particular setof data requested, a file for example, may be divided into a number ofnon-contiguous blocks, requiring the use of devices to reassemble thescattered pieces of data, for example, scatter-gather lists (SGLs) whichmay identify the various media device 17 side addressing and ordering ofdiscontinuous sequences of data.

[0043] In addition to the above-noted address translation process, thestorage processor 15 may perform calculations to support errordetection, error correction, encryption coding, compression, or othertypes of calculations. The results of such calculations may include theappending of data to the user data stored on the media or the morphingof user data to another form or a combination of both. Data derived fromcalculations may be added to other housekeeping data, the combination ofwhich is often categorically referred to as “metadata,” such that theuser data stored on the media devices 17 is different, at least in part,from that passed by the requesting processes 10 to the storage processor15.

[0044] Due to the above addressing translation and calculations as wellas the translation among protocols including their respectivecommunications channels, the storage processor 15 has the potential tomove data very slowly, which is counter the demands of requestingprocesses 10. Delays can manifest due to long latency (slow access andretrieval) or low throughput (low rate of data transfer). To addressthis, the slowest data transfers and/or calculations may be done inparallel by suitable construction of the storage processor 15 andattached systems. But parallel data and computation paths presentordering and coherency issues. These may be addressed by variousfeatures of storage processors, as described above and below, whichinclude judicious choice of processing/transfer routes configured forparallel operation, and several mechanisms for prioritizing and controlof ordering of events.

[0045] Referring again to FIG. 5, when a requesting process 10 makes atransfer request, control data indicating the request is sent to controlmemory 46. The transmission of the request may involve translation andvarious steps such as storing the request in a queue, buffering it,determining its priority, etc. and these may be performed by an externaldevice, for example the host adapter 74 as shown in FIG. 4. The controlprocessor 45 may learn of the request in its memory by various meanssuch as a program loop executed thereon to poll a request queue storedin control memory 46 or by the requesting process 10 invoking aninterrupt signal. The host adapter 74 may generate the latter, forexample through the extant chain of interfaces (e.g., PCI-X bus 72,PCI-X bus interface 70, and FE interface 42).

[0046] The reason for initiating a transfer in this way is so that thestorage processor 16 may handle data that relates to where and how therequested data is to be transferred to the storage processor 16. In theembodiment of FIG. 5, data to be stored is first placed in user memory130 and then processed from there. The control processor 45 may thusrespond to the request by determining what areas of user memory 130 areavailable and what areas should be used to meet the request. Becausenon-contiguous portions of memory may be required to satisfy therequest, the control processor 45 may transmit a scatter-gather list(SGL) (for example by placing in control memory and transmitting apointer or by some other means) to the host adapter 74 to indicate theportions of the user memory 130 that should be used for the transfer. Tominimize the burden on the control processor 45 for this step, thecontrol processor 45 may place the SGL in its control memory 46, fromwhich the host adapter 74 may fetch it.

[0047] Once the host adapter 74 has the SGL, it can then place the datato be stored into user memory 130 as indicated by the SGL and notifiesthe control processor 45 that the data is ready. To minimize the burdenon the control processor 45, in the preferred embodiment, the hostadapter 74 may invoke another interrupt to provide this notification.

[0048] In the preferred configuration of a storage processor 15/16 theabove data transfer does not occur in a serial fashion. Preferably, thestorage processor 15/16 is configured to allow multiple threads to runsimultaneously and for multiple data transfer operations to occursimultaneously. Thus, for example, the host adapter 74 is preferablypermitted by the control and data communications devices 110 and 115, tosend its data to the storage processor 15/16 and then move on to itsnext step. The storage processor may accomplish this by providing in thecontrol and data communications devices 110 and 115 caching orbuffering. The use of caching implies that duplicate copies of data thatcorresponds to certain addresses in memory are established and a set offormal mechanisms to address these. In a preferred embodiment, a simplerscheme is employed in which the data transferred by the host adapter 74is accepted in chunks to allow the host adapter 74 to process otherthreads rather than wait for all the data to flow into the destination.The data paths leading through the control and data communicationsdevices 110 and 115 to the memory 130 may also include buffers to allowsimultaneous data traffic in the storage processor 15, 16 to sharecommon channels. The latter is discussed further elsewhere in theinstant specification. The situation that may exist however is thatafter the final set of data is moved into a buffer in the storageprocessor 15, 16, the host adapter 74 may then invoke the interrupt andmove on with the processing of other threads before the data arrivesfully in the memory 130. This presents an ordering problem. If thecontrol processor 45 begins an operation on the data in the locationswhere the user data are expected to be found, the wrong data will likelybe processed. Interrupts are preferably employed to control the flow ofvarious simultaneously executing threads with buffered data transfersbecause they are inherently minimally burdensome on the requesteddevices, but in they can also lead to ordering problems such asidentified above. These may be addressed as discussed elsewhere in theinstant specification.

[0049] Typically, as discussed previously, the data that is actuallystored to the media devices 17 may take up more memory than the dataoffered up by the requesting processes 10. As may be recognized by thoseof ordinary skill, the transfer of data to contiguous portions of amemory can normally be performed much faster than to many non-contiguousportions of memory. Many storage systems deliberately package data inblocks and pair each to a set of metadata with an original set of dataand an appended set of metadata. For example, the incoming user data maybe divided into 512 byte blocks and 8 bytes of metadata added to each.As the incoming data is stored in memory, the metadata may be generatedand then added to each block so that the memory contains successiveblocks of 520 bytes. When it comes time to transfer the incoming datafrom memory to the media devices 17, the high speed contiguous transferscan be used. The latter situation is called back end friendly.Alternatively, the data can be written to the memory 130 in successive512 byte blocks using high speed contiguous transfers and the metadataappended as the user data is transferred to the media devices 17. Thissituation is called front end friendly. In a preferred embodiment, theSGL used to transfer data into memory 130 (or transfer data out in asimilar process) may be generated such that they separate the incomingdata into blocks such that metadata is in memory. Alternatively, datamay be transferred into and out of memory 130 using high speedcontiguous transfers by padding the 520 byte (520 bytes is used in theexample, but not limiting of the technique) blocks when data istransferred to or from memory 130 from the requesting process 10 side.Either system is inherently back end friendly. Various techniques may beemployed to provide the padding data, such as a front end process thatadd the data to a source memory (not shown) before the transfer to theuser memory 130.

[0050] In a preferred embodiment, upon receipt of the transfer request,the control processor, in addition to supplying a SGL to the requestingprocess host adapter 74, may program the data controller/translator 60to process the data in memory. The process may involve any or all thecalculations mentioned above, for example, error checking data (e.g.,cyclic redundancy check or CRC), implementation of data compressionalgorithms, generation of error correcting data, and addition of recordssuch as so-called time and date-stamping. In the preferred embodiment,the control processor 45 defines multiple instruction queues in controlmemory 46 or user memory 130 and places the instructions for therequested operation in a queue based on a priority assigned to thequeue. When the data controller/translator 60 completes all theinstructions in a current queue, it may interrupt the control processor45 indicating a request for assignment to a next queue of instructionsto execute. By operating in this manner with multiple queues, it ispossible for software run by the processor 45 to prioritize the tasksperformed by the data controller/translator 60. Also, the processor 45may specify in its instructions to the data controller/translator 60what points or upon what events the processor 45 should be interrupted.For example, when the data controller/translator 60 has finishedoperating on data in the user memory 130, the Data controller/translator60 may interrupt the processor, as indicated in a set of instructions inthe queue. Also, the data controller/translator 60 may, by default,interrupt the processor 45 each time it reached the end of a currentinstruction queue. It could then be reassigned to start again with thesame queue (with new instructions) or assigned a different queue.Preferably, at least some of the queues are maintained in control memoryand most preferably all are maintained in control memory. Because thelanguage of the commands generated by the control processor 45 is a highlevel language, the amount of memory space required by the queues can besmall, which means a smaller control memory 46 may be used for thispurpose.

[0051] Preferably, the DMA operates in a pipelined mode at least to theextent of performing a pre-fetch of its successive instruction inparallel with a currently executing instruction. Also, preferably, thecommands transferred to the input queues of the DMA/XOR engine 125 bythe control processor 45, are high level commands to minimize the burdenon the control processor 45. The machine language commands addressed bythese higher level commands may be stored in any suitable memory spaceand preferably a space that is quickly accessible by the DMA/XOR engine125 and which may be addressed by the control processor or other deviceoutside the storage processor 15/16 such that these machine languageinstructions can be redefined as required. In a preferred embodiment,the machine language instructions are stored in a SRAM to support highuser data throughput.

[0052] The memory 130, in the preferred embodiment, may include multipleaddressing and data channels each of whose widths are the same as thoseof the data channels defined by the control and user data communicationsdevices 110 and 115. For example, in an embodiment, these channels are64 bits wide. One way to use the multiple memories and channels is tomove data in synchrony such that the bandwidth of the memory 130 is amultiple of the data channel widths. A preferred way to employ thememories, however, is to spatially demultiplex data transfers to permitparallel operation. For example, read and write operations, through twoseparate channels, may be performed simultaneously. Since the storageprocessor 15/16 supports multiple threads and a high degree ofparallelism in data transfer, such a use of the memory channels ispreferred over synchronous operation with the attending burdens entailedin operating synchronously, such as translating channel widths. Topermit posting and thereby promote continued parallel operations byrequesting devices, preferably, the user memory 130 is equipped withaddressing queues to permit multiple read and write requests to beposted.

[0053] As indicated in FIG. 5, the data controller/translator 60 mayinclude (or be substantially coterminous with) a direct memory access(DMA) engine and XOR engine 125. The latter preferably includes at leastone first-in, first-out (FIFO) buffer configured to generate and hold acomputed XOR value in its memory cells as each data block is transferredthrough it, until it is reset. It can be any width or depth, but in anembodiment, is one bit deep and some number of bits wide for which aparity value is desired. Data may be passed through the XOR FIFO as itis placed in memory, as it is withdrawn from memory, or it may be drawn,computed and the data replaced in memory in an intermediate operation.

[0054] An XOR FIFO may also be employed to accumulate sector parity, forexample, RAID 5 data sectors, may be retrieved through the BE interface48 under control of the DMA XOR engine 125. The first sector of RAID 5storage “stripe” may be fetched from the media devices 17 and placed inthe XOR FIFO. Other sectors from the same stripe may then be fetched andXORed with the previous contents of the FIFO. This process may continueuntil all of the sectors of the stripe have been processed. The finalproduct of the XORs is the parity calculation or re-generation of thedata for a particular RAID 5 stripe. The process of generating theparity sector is analogous.

[0055] The advantage of the XOR FIFO is that a minimal amount of ASIClogic and RAM storage is needed to process the stripe parity calculationor regenerate the stripe data. Also, it is not necessary to fetch allthe stripe's sectors before the parity calculation or data re-generationis started. This allows greater amount a parallelism to exist andtherefore reduces the amount of time it takes to process the datastripe.

[0056] Once the data is in user memory 130 and the DMA/XOR engine 125has finished processing it, the DMA/XOR engine 125 may generate aninterrupt to the control processor 45. At some time beforehand, thecontrol processor 45 may generate a SGL and place it in control memory46 or it may do so in response to the interrupt request from the DMA/XORengine 125. The SGL may be fetched by the BE device adapter 84 from aqueue of transfer requests. The latter may be maintained in controlmemory 46 or in user memory 130 or in another memory (not shown) asdesired. Preferably, the transfer requests are stored in control memory46 to minimize the time burden on the control processor 45.

[0057] Once the device adapter 84 has completed the transfer of datafrom user memory 130 to the media devices 17, the control processor 45may generate an acknowledgement message or information to be included inone and place that in control memory 46. The control processor 45 maythen signal an indication to the device adapter 84 that the informationis ready and the FE host adapter 74 may send the message to therequesting process 10.

[0058] Referring now to FIG. 6A, the control and user datacommunications devices 110 and 115 as well as the FE, BE, and PEinterfaces 100, 135, and 140, constitute a communications fabric thatprovides a store and forward function by means of various bufferingcomponents 251. These buffering components 251 may be configured tosupport priority allocation of communications channels as well as amechanism to permit data to be posted by a transmitting process (any ofvarious devices connected to the fabric) without requiring thetransmitting process to wait. The configuration illustrated at 253 isarbitrary and for purposes of general description only and is notintended to be limiting beyond the discussion that follows. Theregisters 250, 260, 265 introduce an N-clock (preferably single-) cyclestaging component. The multiplexer 258 chooses between two signal paths259 and 261. The signal path 259 contains a buffer 255 to permit data tobe accepted from an upstream process 267 immediately by the secondsignal path 259, even if a downstream process 256 is not ready to acceptdata. The buffer 255 permits the upstream process 267 to post its dataand continue without interruption despite the fact that a shared channel264 may be busy carrying data from another signal path 261. Bypermitting data to be buffered in this manner, priority may be given tothe signal path 261 for the transfer of data. In an embodiment, this maybe used in the environment of a storage processor 15/16 to promote highthroughput by giving priority to user data being transferred between BEand FE. This priority may be granted under hardware or software control.For example, the multiplexer 258 and downstream process 256 may becontrolled such that data on the signal path 259 is always held up inthe buffer 255 until the signal path 261 is cleared. For example, thesignal path 261 may be used to carry user data going between BE and FEand signal path 259 to carry control data, thereby giving priority tothe user data.

[0059] Referring to FIG. 6B, a buffer 272 receives data transmitted froman upstream process 284. The data in the buffer 272 is read by either adownstream process 286 or a buffer 270. The upstream and downstreamprocesses 284 and 286 may include buffers, switches, storage devices,processors, buses, etc. The buffer 270 is contained in a first data pathand the downstream process 286 in a second data path. Registers 285,275, and 280 may be provided to insert a single cycle staging element.

[0060] Either of the embodiments of FIGS. 6A and 6B may be used to helpto insure that shared data paths are flushed of non-critical dataquickly to sustain more critical data, for example, that required tosustain high throughput. As discussed above, that the maintenance ofhigh throughput may be one of the features of the storage processor 15.In the embodiment of a storage processor shown in FIG. 6C, buffers anddata path connections are provided to keep a shared data path 420flushed of control data being transmitted from a FE interface 465 to aPE interface 460. That is, for example, control data may be sent fromthe requesting process 10 to the control processor 45 along the shareddata path 420, which includes a processor end FIFO buffer 410. Theshared data path 420 leads to a control branch 423 that is connectedthrough a data switch 475, with a demultiplexer 425, to a processor endbranch 430, which applies control signals to the PE interface 460.Immediately after a junction 484, on the control branch 423, alook-aside FIFO buffer 405 is configured to read control data from theshared path 420. The effect of this is that the control data does notneed to wait for the data switch 470 and processor end buffer 410 toready to transmit the control data through the control branch 423. Theshared data path 420 also leads to a user data branch 423 that isconnected through the data switch 475, with a demultiplexer 425, tomemories 481 and 482, which applies data signals to the memories 481 and482. By providing the look-aside FIFO buffer 405 in this branchingconfiguration as shown, user data is free to flow into memory along theshared data path 420 and the user data branch 423 without waiting forcontrol data to be accepted upstream. An alternative to the aboveconfiguration is to provide a separate port in the FE interface 465 foruser and control data.

[0061] Note that there may be multiple instances of branches withlook-aside buffers as the FE, BE, and PE interfaces 465, 455, and 460may be essentially identical. Also note that the components shown, whichinclude bus interfaces (e.g., PCI-X cores) on each of the FE, BE, and PEinterfaces 465, 455, and 460, switches, etc. may be formed on a singleASIC for high reliability and speed.

[0062] Referring now to FIGS. 3, 4, 5, and 8A, a first flowchartsummarizes the process of transmitting information through the storageprocessor 15 from the requesting process 10 to the media devices 17 orother devices outboard of the BE interface 48. In a RAID system, thiswould be a process responsive to a write request. In step S10, therequesting process 10 generates a transfer request to store informationon media devices 17. In step S15, the host adapter 74 accepts therequest and places a request in control memory 46. In step S20, the hostadapter generates an interrupt to indicate to the control processor 45that a request is in control memory 46. Next, in step S25, the controlprocessor 45 generates allocation data, for example, an SGL and outputsit to control memory 46. The host adapter 74 then posts the user datafor which the request was generated to the storage processor 16 FEinterface 47. The latter is connected to the backplane communicationsdevice 50 (or the equivalent user data communications device 115), whichas discussed above, provides a store and forward communications fabricthat allows the user data to be offloaded quickly and may take someinterval to be transmitted to its destination(s) in user memory 130.

[0063] In step S35, after completing the posting of user data, the FEhost adapter invokes a maskable interrupt to the control processor 45.The FE host adapter makes no attempt to control the timing of theinterrupt or any ordering issues that arise despite the fact that, asillustrated in step S40, the user data may still be proceeding to usermemory 130 at the time the interrupt is invoked. To prevent theinterrupt from being responded to by the control processor 45 before theuser data is in user memory 130, the interrupt is masked at step S36until it is determined at step S37 that the data has been flushed fromthe data paths 165, which, as discussed above may contain buffers.

[0064] A number of techniques may be used to determine if the data pathis flushed. For example, an end of transmission symbol may be sent asthe last word on the data path and detected when it arrives in usermemory 130. According to a preferred method, the line may be determinedto be flushed by hardware logic. Referring momentarily to FIG. 9, anarbitrary communication path 260 in the control and user datacommunications devices 110 and 115 includes a series of FIFO buffers220, 225, and 230. Data is written to the first FIFO buffer 220 and isread from the last FIFO buffer 230. The communication path 260 isflushed only when the last data word written to the first FIFO buffer220 is read from the last FIFO buffer 230. FIG. 9 shows an algorithmthat may be implemented in software and/or hardware to determine when,upon the posting of the last data word to be operated on in response toan interrupt, when that last word has been flushed from the path 260.Preferably the implementation is done without involvement of the controlprocessor 45 or the DMA/XOR engine 115 (or equivalently, the datacontroller/translator 60). FIG. 9 assumes the FIFO buffers areimplemented with pointers in a memory, which is the most common type ofimplementation, but the equivalent can be implemented using shiftregisters or any other type of FIFO. First, in step S200, the writepointer of the first FIFO buffer 220 is saved in response to theinterrupt. For example, hardware logic may detect the leading edge ofthe interrupt signal and save the value of the write pointer at thatinstant to a memory register. Step S205 is executed when the first FIFObuffer 220 empties (R=W1) of data following the interrupt (target data),the write pointer of the second FIFO buffer 225 may be stored in aregister (not indicated in the drawings). Step S210 is executed when thesecond FIFO buffer empties of the target data and the write pointer ofthe third FIFO buffer 230 is saved to a register. Step S215 executeswhen the third FIFO buffer 230 empties of the target data and the pathis declared flushed at that point. The interrupt mask may then bewithdrawn and the control processor 45 may respond accordingly.

[0065] Note that registers for storing FIFO pointers and/or flags thatmay be used to provide the logical control of FIG. 9 or alternativeembodiments are indicated figuratively as local space 120. The localspace 120 is not necessarily localized as indicated in the figure. Theindication is only an abstraction to show that other storage for variousdata used in controlling the processes of the storage processor 15exist.

[0066] Returning to FIGS. 3, 4, 5, and 7A, the control processor 45 mayqueue translation tasks at step S45 in response to the interruptunmasked at step S42. Referring now also to FIG. 7B, the datacontroller/translator 60 may then perform any required transformationson the user data in user data memory 130. For example, RAID 5longitudinal parity may be calculated and added to the sectors in usermemory 130. Alternatively, rather than storing data in memory and thenperforming translations on it and subsequently returning the translateddata to memory, the data may be moved through a buffer, such as an XORbuffer, and the data translated as it is written to the user memory 130.

[0067] Preferably, data is operated on once it is in memory. Referringbriefly to FIGS. 10 and 11, data blocks or words 320 being transferredto user memory 130 may be transferred through an XOR buffer 310 whichretains the parity of all successive blocks or words transferred throughit until reset. After the final data block of a sector 325 istransferred into memory, the contents of the XOR buffer 310 may betransferred to a memory location and provides the longitudinal parityblock 300 for the sector 325. Then, when data is transferred out of usermemory 130, each block may be transferred through XOR buffer 310 in sucha way as to generate a parity sector 370 by transferring blocks 300/305in order through the XOR buffer 310 (or the order may be taken intoaccount when the data is used or transferred later). That is, a block300 from a first sector 325 may be transferred first as indicated byarrow 351 through the XOR buffer 310 to a destination or back to theuser memory 130 as indicated at 353. Then a block from a next sector(not shown but intermediate between sectors 325 and 327 as indicated byellipses) may be transferred through the XOR buffer 310 to a destinationor back to the user memory 130. When the last sector 327 is reached, ablock from that sector 327 may be transferred through the XOR buffer 310as indicated by arrow 352 to a destination or back to the user memory130 as indicated at 354. After the transfer of a block from the finalsector 327, a parity block for the parity sector 370 may be transferredfrom what remains in the XOR buffer 310 as indicated by arrow 359. TheXOR buffer 310 may then be reset and the process repeated for the nextblock 305 beginning with the first sector 325 as indicated by arrows355, 357, 356, 358. After the next blocks of the last sector 327 aretransferred through the XOR buffer 310, the contents again may be takenas the next block of the parity sector 370 as indicated by arrow 360.The process is repeated for all the blocks of all the sectors until theentire parity sector 370 is completed.

[0068] Note that, rather than create the parity sectors piecemeal asindicated above, a sector-sized XOR buffer (not shown) may be used andthe data transferred sector by sector, rather than block by block/sectorby sector. Also note that other types of error detection and correctioncodes are applicable in this context such as cyclic redundancy check(CRC).

[0069] Returning again to FIGS. 3, 4, 5, and 7B, step S50 may includeother operations on user data such as data compression, encryption,other types of error correcting or checking codes, translation, etc.Next, in step S55, the data controller/translator 60 may interrupt thecontrol processor 45 to indicate that the data is ready to be moved tothe media devices 17 (FIG. 1A) or other destination via the BE interface140. A command to apply the interrupt may be provided in the instructionqueue generated by the control processor 45 as discussed above. Thecontrol processor 45, in response to the interrupt, may output an SGL orother data to control memory 46 at step S60 to permit the BE deviceadapter 84 to retrieve the user data. In step S70, the BE device adapter84 then transfers data from user memory 130 to media devices 17 or otherdevices according to the particular configuration. The BE device adapterthen signals the control processor 45 that it has completed the datatransfer by invoking an interrupt at step S75. The control processor 45,at step S80, may then generate an acknowledge-transfer signal, forexample a message. The signal may be placed in control memory 46. Instep S85, the FE host adapter 74 then fetches, from control memory 46,and transmits the acknowledge-transfer signal to the requesting process10.

[0070] Referring now to FIGS. 3, 4, 5, and 8A, a second flowchartsummarizes the process of transmitting information through the storageprocessor 15 from the media devices 17 or other devices outboard of theBE interface 48 to the requesting process 10. In a RAID system, thiswould be a process responsive to a read request. In step S110, therequesting process 10 generates a transfer request to read informationfrom media devices 17. In step S115, the host adapter 74 accepts therequest and places a request in control memory 46. In step S120, thehost adapter generates an interrupt to indicate to the control processor45 that a request is in control memory 46. Next, in step S125, thecontrol processor 45 generates allocation data, for example, an SGL andoutputs it to control memory 46. The control processor 45 also signalsthe BE device adapter 84 at step S125 to indicate that the data may betransferred to user memory 130. The BE device adapter 84 then posts theuser data for which the request was generated to the storage processor16 BE interface 47. As in the write request discussed with reference toFIGS. 7A and 7B, the latter is connected to the backplane communicationsdevice 50 (or the equivalent user data communications device 115), whichprovides a store and forward communications fabric that allows the userdata to be offloaded quickly and may take some interval to betransmitted to its destination(s) in user memory 130.

[0071] In step S135, after completing the posting of user data, the BEdevice adapter 84 invokes a maskable interrupt to the control processor45. Again, the BE device adapter may make no attempt to control thetiming of the interrupt or any ordering issues and the interrupt may bemasked until the data path is flushed. The flushing of the data path maybe determined in step S137 in various ways as discussed above withrespect to the write request operation.

[0072] The control processor 45 may queue translation tasks at step S145in response to the interrupt unmasked at step S142. Referring now alsoto FIG. 7B, the data controller/translator 60 may then perform anyrequired transformations on the user data in user data memory 130. Againthe transformation may be handled in various ways as discussed withregard to the write request operation. For example, the data in thememory 130 may be used to regenerate bad sectors, as in RAID 5.Referring now to FIGS. 3, 4, 5, and 8B, step S150 may include otheroperations on user data such as data compression, encryption, othertypes of error correcting or checking codes, translation, etc. Next, instep S155, the data controller/translator 60 may interrupt the controlprocessor 45 to indicate that the data is ready to be moved to therequesting process 10 (FIG. 1A). A command to apply the interrupt may beprovided in the instruction queue generated by the control processor 45as discussed above. The control processor 45, in response to theinterrupt, may output an SGL or other data to control memory 46 whichmay be read by the FE host adapter 74 at step S160 to permit it toretrieve the user data. In step S165, the FE host adapter 74 thentransfers data from user memory 130 to the requesting process 10. The FEhost adapter 74 then signals the control processor 45 that it hascompleted the data transfer by invoking an interrupt at step S175. Thecontrol processor 45, at step S180, may then generate anacknowledge-transfer signal, for example a message. The signal may beplaced in control memory 46. In step S185, the FE host adapter 74 thenfetches, from control memory 46, and transmits the acknowledge-transfersignal to the requesting process 10.

[0073] Preferably, in the above embodiments, separate buffers areallocated for read and write transactions. This allows write posting ofdata to be done while a particular read transaction is being processed.Without this feature, the write transaction has to wait until the readtransaction is processed before it may proceed. Note also that in theabove embodiments, the control processor is not informed of changes toany temporary copies of data as would be required if caching were usedin routing the data between FE, BE, and PE. Caching may be provided foruser data stored in user memory 130, but this, in a preferredembodiment, is non touched by the processor at all; that is, theprocessor may not be informed of changes to data in the user memory 130.This keeps the control processor from being burdened by invalidates.Note that although the above embodiments contemplated the use of an ASICfor implementing the control processor 15/16, many of the features ofthese embodiments may be implemented using commodity devices. Forexample, the data controller/translator 60 may be instantiated as amicroprocessor for performing the specialized functions required by thestorage processor 15/16. That is, there would be two independentprocessors, one for control and one to handle the user data andtranslation operations.

[0074] Certain benefits are achieved by means of the above embodiments.These include:

[0075] 1) User data may be placed in memory in a single step to generateparity data such that user data is transferred from disk to memory,parity added to user data in memory, and parity and user datatransferred from memory to client process in no more than two steps withintervening steps for generating parity data.

[0076] 2) The translation between protocols from disk to memory and backto the requesting process may provide compatibility with standard busprotocols.

[0077] 3) Address translation is handled on the SP to unburden FE & BE.There is no need for either of them to handle scatter-gather data.

[0078] 4) The memory controller and XOR engine are integrated in samedevice which:

[0079] a) permits look-ahead requests and

[0080] b) The XOR engine can be combined with the DMA.

[0081] 5) User data need not be “snooped” on controller processor bus.

[0082] 6) The code used by the DMA/XOR engine may be reprogrammed.

[0083] 7) The DMA sends an interrupt to processor when its queue runsout, rather than pushing its status. The processor can then keep the DMAworking rather than getting around to looking at its status.

[0084]FIG. 10 illustrates steps involved in handling a single writerequest in a RAID system in which longitudinal and non-longitudinalparity are generated to complete the handling of the request. Thesituation illustrated is where only one data block is overwritten. Fourblocks of data are involved: an old parity block 515, which comes fromthe BE, particularly, one or more disks, depending on the details of theRAID implementation; an old block 520, i.e., a copy of the block to bewritten over; a new block 525, i.e., the new data block being written tothe disks which comes from the host; and a new parity block 530 which iscalculated from the foregoing data blocks 515, 520, and 525. Tocalculate the new parity block 530, old parity block 515 is read intothe XOR FIFO 505 as indicated by arrow 515, then the old block 520 isread in as indicated by the arrow B to reverse its contribution to theold parity block 515. Then the new block 525 is read into the XOR FIFO505 as indicated by the arrow C and the new parity block 530 read outand stored in user memory 540 as indicated by the arrow D. Note that thehost may convey a portion of a new block and this could be written overcorresponding portions of the old block 520 as would be understood bypersons of skill in the art.

[0085] If the process of FIG. 10 were done over and over again inmultiple threads (assuming level allocation of the independent memorychannels 481 and 482—FIG. 6C), the situation of illustrated in FIGS. 11Aand 11B would result. That is, data would stream from the back end 550into memory 540 as indicated by arrow H; data would stream through theXOR FIFO 505, from and to user memory 540, as indicated by arrows E andG. Data would stream from the FE 560 as indicated by arrow J and resultswritten to the BE 550 as indicated by arrow F. As illustrated in FIG.11B, transfers corresponding to arrows E, F, G, H, and J may occuressentially simultaneously. Note that the use of the two memories 481and 482 as shown in FIG. 6C is particularly significant forload-leveling in this context if the memory speed is such that a singlememory channel (e.g., bus) is unable to handle the throughput potentialof the FE and BE interfaces 47 and 48 (FIG. 3). (See discussionattending FIGS. 13A and 13B). Some of the lines (E, G, and H) in FIG.11B are broken to illustrate that maintenance of continuous saturationof the bandwidth of the FE and BE interfaces 47 and 48, which areillustrated by lines F and J, may not require continuous transfer ofdata represented by these lines (E, G, and H). This assumes the size ofthe channels carrying the data transfers represented by each line E, F,G, H, and J is the same bandwidth and, of course, this need not be thecase and is shown just for illustration of the basic concepts discussed.

[0086] Referring to FIGS. 12A and 12B, a read request withreconstruction of lost data from non-longitudinal parity (parity blocks)may be satisfied with full saturation of the FE and BE 47 and 48channels 47A and 47B in a similar manner as discussed in connection withFIGS. 11A and 11B. In FIGS. 12A and 12B it is assumed multiple threadsof just read requests are being satisfied. Data streams from the backend 550 into memory 540 as indicated by arrow M; data streams throughthe XOR FIFO 505, from and to user memory 540, as indicated by arrows Land K. Data streams to the FE 560 from the memory 540 as indicated byarrow N. As illustrated in FIG. 12B, transfers corresponding to arrowsK, L, M, and N may occur essentially simultaneously. Again, since thereare shown simultaneous reads from memory, it is assumed that there islevel allocation of the independent memory channels 481 and 482 shown inFIG. 6C.

[0087] Referring to FIGS. 13A and 13B, with two memories (e.g., 481 and482) with respective access channels, e.g., high speed memory buses, theuser data channels from FE and BE may be saturated for all datatransfers between FE and BE. To saturate the FE channel, the number ofmemory channels and their bandwidths must sufficient to provide therequired number of simultaneous reads and writes. Referring to FIGS. 13Aand 13B many read and write requests multi-threaded so that they occurnearly simultaneously permitting, approximately, full saturation of theFE and BE channels. Each transfer through the FE or BE may requiremultiple reads and writes from memory. Thus, the memory bandwidthmultiplied by the number of memories and memory channels must be greaterthan three times the FE bandwidth to provide for the multiple memoryaccesses required. But with suitable load balancing, the bandwidth ofeach memory can be reduced below that required if a single memory andchannel were used. The load balancing among the multiple memory writes,represented by the lines P and Q, and the multiple simultaneous memoryreads, represented by lines R and S, is assumed. These pairs of datastreams are indicated at 570 and 580. Although all the lines P-U in FIG.13B are shown as unbroken, suggesting that all channels associatedtherewith are saturated, it is obvious that this is an idealizationsince not all operations would stream through the XOR FIFO 505.

[0088] Note that the preceding discussion of FIGS. 10-13B is somewhatidealized since in the embodiments discussed above, there may be someattenuation of throughput caused by sharing of channels such asdiscussed in connection with FIG. 6C.

[0089] Although the foregoing invention has, for the purposes of clarityand understanding, been described in some detail by way of illustrationand example, it will be obvious that certain changes and modificationsmay be practiced that will still fall within the scope of the appendedclaims. For example, the devices and methods of each embodiment can becombined with or used in any of the other embodiments.

What is claimed is:
 1. A storage processor for a RAID system,comprising: a communications device including an ASIC with a switchconnecting a front end (FE) connectable to a requesting device through aFE bus, a back end (BE) connectable to a disk array through a BE bus,and a processor end (PE) connected to a control processor and controlmemory through a processor bus; said communications device connecting atleast two memories, a DMA engine, a data translator, and a crossbarswitch; said communications device defining data channels lying withinand outside said switch; at least some of said data channels conductingboth user and control data along portions lying outside said switch. 2.A storage processor as in claim 1, wherein said FE, said BE, and said PEinclude identical numbers of channels connecting said FE, BE, and PEbuses to said data channels defined by said communications device.
 3. Astorage processor as in claim 2, wherein interfaces connecting said FE,BE, and PE buses to said data channels include identical numbers ofbuffers.
 4. A storage processor as in claim 1, wherein interfacesconnecting said FE, BE, and PE buses to said data channels includeidentical channel layouts.
 5. A storage processor as in claim 1, whereinat least said user data channels are bidirectional full duplex.
 6. Astorage processor as in claim 1, wherein said at least some of said datachannels are connected to output ports of said switch and to a sharedchannel through a multiplexer, at least one of said output ports beingconnected to a buffer located outside said switch.
 7. A storageprocessor as in claim 1, wherein said at least some of said datachannels are connected to input ports of said switch and to a sharedchannel through a multiplexer, at least one of said input ports beingconnected to a buffer located outside said switch.
 8. A storageprocessor as in claim 1, wherein data translator includes a paritycalculator.
 9. A storage processor as in claim 1, wherein said switchincludes a crossbar switch.
 10. A storage processor for a RAID system,comprising: a communications device including an ASIC with a switchconnecting a front end (FE) connectable to a requesting device through aFE bus, a back end (BE) connectable to a disk array through a BE bus,said communications device having a control processor and controlmemory; said communications device connecting at least two memories, aDMA engine, a data translator, and a crossbar switch; saidcommunications device defining bidirectional (full duplex) user datachannels lying within and outside said switch; at least some of saiddata channels conducting both user and control data along portions lyingoutside said switch.
 11. A storage processor as in claim 10, whereinsaid FE and said BE include identical numbers of channels connectingsaid FE and BE buses to said data channels defined by saidcommunications device.
 12. A storage processor as in claim 11, whereininterfaces connecting said FE, and BE buses to said data channelsinclude identical numbers of buffers.
 13. A storage processor as inclaim 10, wherein interfaces connecting said FE and BE buses to saiddata channels include identical channel layouts.
 14. A storage processoras in claim 10, wherein said control processor and control memory areconnected to said communications device through a processor bus.
 15. Astorage processor as in claim 10, wherein said at least some of saiddata channels are connected to output ports of said switch and to ashared channel through a multiplexer, at least one of said output portsbeing connected to a buffer located outside said switch.
 16. A storageprocessor as in claim 10, wherein said at least some of said datachannels are connected to input ports of said switch and to a sharedchannel through a multiplexer, at least one of said input ports beingconnected to a buffer located outside said switch.
 17. A storageprocessor as in claim 10, wherein data translator includes a paritycalculator.
 18. A storage processor as in claim 10, wherein said switchincludes a crossbar switch.